Deformable 3D Reconstruction with an Object Database

Deformable 3D reconstruction from 2D images requires prior knowledge on the scene structure. Template-free methods [1, 2, 5, 6, 9, 14] use generic prior knowledge such as piecewise smoothness but require multiple images with significant baseline. Template-based methods [4, 10, 13] require only one image but handle only one object for which they need specific prior knowledge, namely a 3D template. In this paper, we propose a novel method that alleviates the strong assumptions of both the template-free and template-based methods: our method uses multiple templates to achieve deformable 3D reconstruction from only one image and for multiple objects. It uses object recognition to automatically discover what objects are visible in the input image and to select the appropriate templates for deformable 3D reconstruction. The object database is built offline. Crucially, this database does not only contain appearance descriptors as in existing object recognition frameworks [7, 8, 11], but also material properties to facilitate deformable 3D reconstruction.